6.835 Intelligent Multimodal User Interfaces

This paper presents a new paradigm for “time-based-information” interactions. The system, called a WallComputer, provides similar ease of use and the rapid accessibility of devices such as calendars and clocks, while also providing a rich and extensive set of interactions generally only found on a multipurpose PC. The presented system incorporates two input modalities: gesture and speech. I present the design of a gesture recognition system that is able to achieve over 99% accuracy, and the design of a contextual-booster that dynamically restricts the command set of each modality. Finally, I demonstrate that dynamic command-set-restrictions allow the system to achieve superior performance.